got this to the state that is as close to go-libp2p release as possible, filling this here so I won't forget:

• remove from all go.mod files and switch to go-libp2p release

There seem to be a regression between 9b887ea (this PR) and 433444b (latest master) caused by TCP reuse logic in go-libp2p bumped here.

I did curl localhost:5001/debug/pprof/profile > ipfs.cpuprof and CPU profile seems to confirm the source is TCP reuse from libp2p/go-libp2p#2984? (cc @aschmahmann @MarcoPolo):

How to reproduce

• Start ipfs daemon
• Open http://127.0.0.1:5001/webui#/peers and wait a minute:
  • this will trigger connections to peers to fetch any missing block get for geoip data (if not cached locally) – acting like accelerator for any underlying bugs in TCP port reuse
  • 💢 staging-2024-12-04-9b887e (go-libp2p master v0.37.1-0.20241202220543-9024f8e8c86e) will grow CPU use once ipfs-webui Peers screen is opened. CPU peaks within 1minute.
  • 💚 master-2024-12-03-433444b (go-libp2p v0.37.2) does not trigger high CPU load

Here are docker images for repro convenience:

• docker run --rm -it --net=host ipfs/kubo:staging-2024-12-04-9b887ea
• docker run --rm -it --net=host ipfs/kubo:master-2024-12-03-433444b

Related work:

• fix(sampledconn): Correctly handle slow bytes and closed conns libp2p/go-libp2p#3080
• refactor(sampledconn): Add Windows OSPeekConn impl libp2p/go-libp2p#3081

libp2p/go-libp2p#3080 seems to fix the CPU spin.

Tested in 2adb2f1 – need to investigate why CI fails.

I am debugging sharness with make -O -j 1 because CI runs with -j 10 and logs are mangled.

Getting bizzare. Tests with the same go-libp2p and boxo version pass in #10631.

The only go.mod diff between a436f4e (this PR, ci failing due to timeout after 20m) and 070e6ae (#10631, ci green, sharness run takes 5minutes), is:

-       github.com/ipshipyard/p2p-forge v0.1.0
+       github.com/ipshipyard/p2p-forge v0.2.0

other than that, we don't change anything in default config other than enabling shared tcp.
Will disable (b5cfd6d) and re-run, to see if that causes issue somewhere.

💡 👉 Sharness passes green if we disable libp2p.ShareTCPListener() (b5cfd6d passed in job here).

Something related to libp2p.ShareTCPListener() makes sharness hang and timeout if run in 10 parallel threads (cd test/sharness and time make -O -j 10).

Still looking for the cause, but I was able to reproduce locally by using the same settings as CI:

$ export TEST_DOCKER=1
$ export TEST_PLUGIN=0
$ export TEST_FUSE=0
$ export TEST_VERBOSE=1
$ export TEST_JUNIT=1
$ export TEST_EXPENSIVE=1
$ export IPFS_CHECK_RCMGR_DEFAULTS=1
$ export CONTINUE_ON_S_FAILURE=1
$ time make -O -j 10 test_sharness coverage/sharness_tests.coverprofile test/sharness/test-results/sharness.xml

# [...]
ok 1717 - add a few entries to big_dir/ to retrigger sharding

expecting success:
    kill -0 $IPFS_PID

ok 1718 - 'ipfs daemon' is still running

expecting success:
    test_kill_repeat_10_sec $IPFS_PID

ok 1719 - 'ipfs daemon' can be killed

# passed all 1719 test(s)
1..1719

# hangs..

It hanged for a few minutes and finished eventually, but took 3x longer than regular run (15m vs 5m).
Re-run and it hanged for 20m and i had to kill it, so there is some factor of randomness to the hang.

Looking at process tree spawned by make, the t0250-files-api.sh and t0260-sharding.sh t0181-private-network.sh t0182-circuit-relay.sh and t0320-pubsub.sh seem to hang the longest in semi-random order.

Will resume tomorrow unless someone else has any idea.

In the meantime, i've re-run https://github.com/ipfs/kubo/actions/runs/12420930073 to double-confirm disabling libp2p.ShareTCPListener() fixes sharness.

Run two tests to narrow down the search:

• 🟢 7a4ec80 disabled libp2p.ShareTCPListener() and sharness finished in 8m successfully (log)
• 🔴 858e10a enabled libp2p.ShareTCPListener() and sharness timeouted after 20m (log)

Maybe increasing timeout would help, but taking 2x as long feels like a bug.

Ok, running sequentially (#10565 (comment)) found the problem:

• Enabling libp2p.ShareTCPListener() break PNET and ./t0181-private-network.sh hangs – kinda makes sense, PNET is TCP-only feature of libp2p and we mess up with TCP.

I'll now confirm this is the only one failing,
and if other tests pass, will disable TCP sharing when PNET is enabled for now.

I'll also see if we can set timeout per test suite / unit, to avoid this debugging horror in the future.

Good news, PNET was the only reason sharness failed. Disabled port sharing when PNET is enabled (bb87df3) and CI is green again:

• https://github.com/ipfs/kubo/actions/runs/12435566625 passed.

I'm going forward with Kubo 0.33.0-rc1.